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Introduction 

National bodies responsible for allocating funds for basic research 
differ in the manner in which they employ peer review (see Chubin 
& Hackett, 1990). Some funding agencies, such as the Research 
Councils in the United Kingdom and the National Institute of Health 
in the United States, establish expert panels for each of a large 
number of narrowly defined research specializations. Applications 
for funding within each specialization are reviewed only by panel 
members. Other agencies also employ a panel system, but decisions 
as to which applications will be funded typically are made only after 
the panel has considered evaluations provided by reviewers who are 
not members of the panel. The National Science Foundation in the 
United States and the Australian Research Council employ this latter 
approach, although in different ways. 

This article is concerned with the place of peer review in the 
process by which the Australian Research Council (ARC) allocates 
funds under the Large Grants Scheme. Awards under the Australian 
Research Council Grants Scheme constitute the primary source of 
funding for basic research in Australia in disciplines other than 
medicine and dentistry (where the National Health and Medical 
Research Council is the responsible body). Policy directives for 
higher education issued by the Australian Government in 1988 (see 
Dawkins, 1988), including centralizing control through a process 
known as “clawback” over funds once distributed by universities, 
increased the influence of the ARC. In 1992 the ARC had a research 
funding budget of $255.5 million, of which $96.1 million were 
committed to research grants, $55 million to postgraduate awards, 
$51.6 million to research infrastructure, $20.2 million to special 
research centres and key centres, and $17.7 million to research 
fellowships. Although universities are responsible for awarding 
research funds under the Small Grant Scheme ($15.7 million were 
distributed in 1992 to 1412 applicants, with a success rate of 33%), 
the primary allocation of the $96.1 million is undertaken by ARC 
through the Large Grants Scheme. Only 19% of all applications for 
an initial Large Research Grant to commence in 1993 were success¬ 
ful, in comparison to success rates of 29% for 1992 and 31% for 
1991. This circumstance arose because the number of applications 
for initial support increased sharply at a time when funding overall 
remained stable and the ARC was committed to provide continuing 
support for projects funded in earlier years. 

An investigator seeking a Large Research Grant lodges an appli¬ 
cation which is evaluated by a discipline panel or a priority panel 
established by ARC. Under the present system each application that 
survives initial culling is sent by the panel to four assessors for 
evaluation. As well as having access to such peer review, the panel 
invites the 40% of applicants surviving the second culling to 
comment on assessors’ reports before deciding which projects 
should he funded. Wilson (1990), when chairperson of the Research 
Grants Committee, the body within the ARC responsible for funding 
under the Large Research Grants Scheme, noted, “The considerable 
increase in competition for research funding by Australian academ¬ 
ics means it is now even more important that the allocation of 


research grants by the Australian Research Council is both fair and 
seen to be fair” (p. 16). Supplementing expert opinion obtained 
through peer review with feedback from applicants might seem to 
provide panel members with the information needed to allow appli¬ 
cations to be assessed strictly on the basis of merit. However, there 
are complex issues relating to peer review and decision making by 
panel members that need to be examined in considering how Large 
Research Grants are allocated by the ARC. 

The questions of particular concern are first whether peer review 
as employed by the ARC is as fair, valid, and reliable as it might be, 
and second what the nature of decision making by panels is. 
Although some of these issues have been a matter for discussion 
within the ARC, there has not yet been independent investigation of 
the manner in which the ARC operates. My objective is to discuss 
questions bearing on the validity and reliability of evaluation of 
grant applications by the ARC with reference to literature on peer 
review and how granting agencies with a panel system of the type 
employed by the ARC operate. As an initial step, however, the 
procedures that the ARC follows need to be outlined. 

The Large Grant Scheme 

Individuals as well as research teams can apply for ARC funding. 
The following outline refers to practices the ARC will adopt in 
evaluating applications for support in 1995 under the Large Re¬ 
search Grants Scheme (see Australian Research Council, 1993). 
Applications close on 28 February 1994. On the basis of previous 
years, 2500-3000 applications are likely to be received. Each appli¬ 
cation will processed by one of nine discipline panels (Biological 
Sciences - Molecular, Biological Sciences - Plant/Animal, Chemi¬ 
cal Sciences, Earth Sciences, Engineering Sciences - 1, Engineering 
Sciences - 2, Physical Sciences, Humanities, Social Sciences) or by 
one of five priority panels (Materials science and mineral process¬ 
ing, Cognitive science. Biology of sustainability. Citizenship, Aus¬ 
tralia’s Asian context). The panels will reject 30% of applications in 
April 1994 without calling for reports from external assessors, and 
a further 30% in July 1994 after reports have been obtained from 
external assessors. The assessors will typically include at least one 
person from among the three nominated by the applicant as qualified 
to evaluate the project. The 40% of applicants who remain in July 
after the second culling will be allowed to provide the panel with a 
one page comment on the assessors’ reports. In September 1994 each 
pane! will generate a rank ordering of all applications still under 
consideration, and assign a score to each application. Lists prepared 
by the discipline panels and the priority panels will go to the 
Research Grants Committee (a constituent committee of the ARC), 
which provides advice to the ARC. Recommendations are then 
forwarded to the Minister for Higher Education and Employment 
Services. Only 19% of initial projects to commence in 1993 and 22% 
of those to commence in 1994 received funding, and in many of these 
cases the level of support was below that sought by the applicant. 
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In S993 each assessor approached by the ARC to evaluate an 
application for a Large Research Grant was requested to report on 
“the quality of the researcher or research group" and “the excellence 
of the proposed research, and the likelihood that it will lead to a 
significant conceptual advance; and/or an important discovery or 
innovation or to the solution of an important practical problem; and/ 
or economic or social benefit for Australia”. Additional criteria 
specified for priority areas were “the intrinsic merit of the proposal”, 
“the probable impact of a successful outcome for the research”, “the 
extent to which the research will lead to collaboration between 
institutions, or groups within institutions”, and “the commercial 
potential or utility of the proposal”. In addition to providing com¬ 
ments on the application, an assessor had to identify the “relative 
ranking of the project and the chief investigators in the international 
community of scholars working in this field” on seven-point scales 
ranging from “bottom 50%” to “top 2%”. Assessors also rated the 
project on six dimensions (originality of the research; soundness of 
planning and methodology; scientific/theoretical merit; technologi¬ 
cal/applied merit, if relevant; feasibility of project; track record of 
the applicant) using a six-point scale ranging from “deficient” to 
“outstanding”. Assessors were also requested to comment on progress 
to date, as detailed in the application, and to identify questions which 
the ARC should raise with the applicant. 

It seems appropriate that ARC panels should obtain peer review of 
applications, since panel members often have to make decisions 
about projects in research areas where they themselves lack expert 
knowledge. However, several issues relating to peer review need to 
be considered. These relate to the selection of assessors by ARC 
panels, the level of expertise of assessors, the extent to which there 
is agreement between assessors who evaluate the same application, 
and the use that ARC panels make of the information they obtain 
from assessors. 

Validity and reliability of peer review 

A recent survey identified concerns about the expertise and 
appropriateness of assessors chosen by ARC panels. Wood, Meek, 
and Harman (1992) asked a sample of applicants who had not been 
funded by the ARC to comment on different aspects of the grant 
review process. Although most respondents favoured peer review as 
a process, many expressed concern about the quality of the reviews 
their application had received, the extent to which assessors chosen 
by ARC seemed to lack expert knowledge, and inconsistency in the 
evaluations offered by different assessors. The survey undertaken by 
Wood et ai. was limited to unsuccessful applicants. Two studies (one 
concerned with the National Science Foundation and the other the 
National Cancer Institute) that took outcome into account found that 
successful applicants were more positive than unsuccessful appli¬ 
cants about the validity of evaluation, including peer review (see 
Chubin & Haekett, 1990). Applicants who were funded by the ARC 
are thus probably more likely than applicants who were not funded 
to claim that the peer review system as employed by the ARC yields 
expert appraisal of applications. 

The Australian Research Council has a moral responsibility to 
ensure that all applications are subject to expert evaluation. To date 
the ARC has not demonstrated accountability on this matter. It is 
impossible to establish from the public record how ARC panels 
select assessors to evaluate applications, and whether there are 
internal checks and balances to ensure that an assessor is expert, not 
open to bias (positive or negative) as a result of prior association with 
the applicant, and free from conflict of interest (such as through also 
being an applicant for ARC funding). The appeal process introduced 
by the ARC in 1992 explicitly excludes appeals "... against the peer 
review process itself, or of the judged merit of the proposal” (ARC 
News , 1992, p. 4), as though issues such as the expertise or compe¬ 
tence of the assessors selected by the ARC or the ARC panel is 
beyond question. Instead, “An applicant should appeal only when 
there is clear evidence that the selection procedures followed have 
contravened the Program Guidelines” (Australian Research Coun¬ 
cil, 1993, p. 24). 
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flic unsui-.ccssful ARC applicants surveyed by Wood et al. (1992) 
sometimes made reference to “luck of the draw”, as though the 
selection of'other assessors would have yielded a different outcome. 

1 here is now an extensive literature, covering review of manuscripts 
submitted to journals as well as evaluation of applications for 
research funding, demonstrating that assessors who are asked to 
evaluate the same material often disagree in the ratings they provide 
(see Cicehetti, 1991). Although the basis for limited consensus in 
evaluation may he that the assessors differ in expertise, in diligence 
in undertaking the task of assessment, or in the standards they 
employ, an alternative is that assessors evaluate the same material 
validly but on quite different bases, Fiske and Fogg (1990) under¬ 
took a content analysis of narratives provided by reviewers who 
evaluated manuscripts submitted to major psychology journals 
They concluded, “In the typical case, two reviews of the same paper 
had no critical point in common. It seemed that reviewers did not 
overtly disagree on particular points; instead they wrote about 
different topics, each making points that were appropriate and 
accurate. As a consequence, their recommendations about editorial 
decisions showed hardly any agreement” (p 591). 

Cole, Cole, and Simon (1981) provided a compelling demonstra¬ 
tion of the importance of selection of assessors. They submitted 150 
applications (75 of'which had been funded by NSF and 75 not funded 
by NSF) to assessors (12 each application) who were expert in the 
field of the research but had not been involved in the original 
evaluation. Many of the applications that had not been funded by 
NSF were rated more highly on this second round ofassessment than 
those that had been funded If support by NSF were dependent solely 
on ratings provided by assessors. Cole et al. estimated that outcomes 
would have been reversed in 25% of cases. Since most ARC 
applications are reviewed by only four assessors, "luck of the draw” 
in the selection of assessors may be a substantial factor in outcome. 

It is a simple matter to determine the extent to which assessors 
evaluating the same application (or the same investigator) agree in 
the ratings they provide. Although such statistics have been calcu¬ 
lated by ARC, they are never made public. It would be surprising if 
the consensus among assessors of ARC applications is higher than 
the low level of agreement reported for peer review in many studies 
in the literature (see Cicehetti, 1991). Since assessors of research 
applications and manuscripts submitted to journals generally dem¬ 
onstrate higher agreement in providing negative than positive evalu¬ 
ation (see Cicehetti, 1991), numerical ratings obtained through peer 
review may provide a reliable basis for ARC panels to identify 
projects that are well below the cut-off point. A number of applica¬ 
tions culled by mid-year as not meriting support probably would 
have received low ratings even if they had been sent for evaluation 
to persons other than the assessors selected by the ARC However, 
the extent to which peer review provides a basis for selecting among 
applications that survive culling is questionable. 

Since attenuation in range lowers consensus in evaluation (see 
Hargens & Herting. 1990). it is unlikely that there would be even 
moderate agreement between assessors among the 40% or so of ARC 
applications that remain following the initial and mid-year cull. 
These presumably all were applications that attracted strong support 
from at least some assessors. The task of panels is to rate the non- 
culled applications so that these applications can be ranked in the 
order in which they merit funding. As noted earlier, only 19% ot 
applications were funded for 1993 and 22% for 1994. It the ranking 
of applications (and hence funding decisions) corresponded to the 
average of the ratings provided by assessors, the question of selec¬ 
tion of assessors would be of vital importance. Following Cole, 
Cole, and Simon (1981), it might be expected that the average of the 
ratings provided by the assessors of a non-culled project would 
change (and in some cases markedly) if a different set of persons 
were recruited as assessors. 

If ARC panels relied simply on the average of ratings provided by 
the four assessors to whom an application is sent, it is likely that 
many worthy projects would not be supported while a number ot 


projects of lesser merit would be funded. There would be a substan¬ 
tial chance component in whether an application is funded. As noted 
by Stinchcombe and Ofshe (1967) in relation to editorial decisions, 
under circumstances where there is a low overall success rate, 
limited agreement between assessors, and decisions reflect the 
average of assessors’ ratings "... it does not take a conspiracy theory 
of journal editing to account for the rejection of a great many good 
papers and the publishing of a large number of mediocre papers” (p. 
117). However, ARC panels do not reach decisions about which 
projects to fund by averaging the ratings assessors provide for each 
application. Instead "... Panels must rate applications in relation to 
one another, which assessors cannot do because they see one or, at 
most, a few proposals. So Panel ratings often differ from the assessor 
average; occasionally they may be lower, or higher, than any of the 
assessors” (ARC News, 1993, p.4). 

The decision making process 

The processes underlying funding decisions by National Science 
Foundation (NSF) panels have been subjected to independent scru¬ 
tiny. Cole and Cole (1981) found that although for some disciplines 
there was a close association between whether a project was funded 
and the average of the ratings provided for the project by assessors, 
funding decisions in other disciplines were not strongly related to the 
evaluations given by assessors. Panels thus decided to fund some 
projects that assessors had not supported strongly, while other 
projects that were evaluated in favourable terms by assessors were 
not funded. 

Klahr (1985) examined the relative influence of evaluations 
provided by panel members and assessors on outcome through 
statistical analysis based on how 200 NSF applications in a single 
discipline (psychology) were processed. The decision making proc¬ 
ess required panel members to rank applications twice, once before 
assessors’ ratings and reports were available and then after access to 
assessors’ ratings and reports. Klahr found that assessors overall 
were lenient in the sense of favouring support at a level well above 
that which NSF was able to fund. In determining which projects to 
fund, the panel did not rely simply on the average of the assessor 
ratings provided for each project. In fact the average of assessor 
ratings correlated only .61 with outcome (whether or not a project 
was funded), and the average of assessor ratings provided only a 
minor contribution to outcome over and above the evaluations that 
the panel members had made before reading the assessors’ reports. 
Klahr argued that the contribution of assessors comes from the 
qualitative (and expert) information they offer panel members, 
rather in the quantitative ratings they provide. He concluded, “The 
ad hoc reviewer's role should be construed more as expert testimony 
and less as judgment. Ad hoc reviewers should continue to be 
instructed to comment on the strengths and weaknesses of proposals, 
but they should be informed that their final overall rating is not as 
important as their substantive comments” (Klahr, 1985, p. 153). 

It would be interesting to know the correlation between the rank 
ordering of applications produced by each ARC panel and the ratings 
given to these applications by assessors. A high correlation would 
indicate that funding decisions closely follow ratings provided by 
assessors, with panel members having limited moderating influence 
on outcomes. However, a low or moderate correlation would suggest 
that evaluations by panel members have substantial impact on 
outcome over and above the ratings provided by assessors. Since 
panels have the right to disregard ratings provided by an assessor 
(see ARC News, 1993, p.4), the above correlations need to be 
calculated on the basis of the ratings that assessors did in fact 
provide. 

The interview process employed prior to 1994 provides prima 
facie evidence that ARC panels treat assessors as expert witnesses 
rather than as judges. The task of an ARC panel is seen as not simply 
averaging asssessors’ ratings, but reaching an informed decision 
based on all relevant evidence including the ratings and qualitative 
comments provided by assessors. In operating on this basis the panel 


can discount the opinion of assessors who are perceived as incom¬ 
petent, uninformed, or biased, identify and msolve bases for differ¬ 
ences in evaluation between assessors, and reconcile what at one 
level may seem to be inconsistent or unreliable evidence. A panel 
functioning in this way operates as jury and judge, with assessors 
treated as expert witnesses (or perhaps sometimes as hostile or 
incompetent witnesses). 

Each ARC panel covers a range of disciplines. For example, in 
terms of disciplinary affiliation the 11 members of the Social 
Sciences and Humanities panel in 1992 included two from psychol¬ 
ogy and one from each of anthropology, classics, economics, educa¬ 
tion, English, geography, history, journalism, and lav/. It would be 
interesting to establish the extent to which matters such as selection 
of assessors and determining the final rating given to a project are 
arrived at by disciplinary representatives rather than by the ARC 
panel as a whole. In 1994 panels will have extended membership, 
although seemingly only for the initial culling at the April meeting 
(see Australian Research Council, 1993, p. 2). 

If individual members play a major role in deciding which 
applications within their own discipline will be funded, the judg¬ 
ment and integrity of the persons appointed to panels is of primary 
importance. Members of ARC panels are appointed for three year 
terms, and typically serve for no more than six years continuously. 
Selection is by a committee within the ARC, but universities, 
industry, the national academies, and other organizations can pro¬ 
vide nominations for vacant positions. The fact that panel members 
often themselves are applicants for an ARC grant, and hence 
competing for a share of the limited funds distributed by ARC, has 
been a matter of comment within the Australian academic commu¬ 
nity. However, the ARC employs procedures to ensure panel mem¬ 
bers are not involved in evaluation of their own applications (see 
Australian Research Council, 1993, pp. 24-25). It nevertheless 
would be interesting to compare the success rate of ARC panel 
members with the success rate of applicants in general. Such data 
have never been published. 

Whereas ARC panels have access to peer review (although typi¬ 
cally from far fewer assessors per application than is the case within 
NSF), there is no requirement that decisions about funding made by 
ARC panels reflect peer review. In contrast to independent scrutiny 
(e. g., Cole & Cole, 1981; Klahr, 1985) of the processes by which 
NSF panels reach funding decisions, the manner in which ARC 
panels operate has not been open to investigation. A useful first step 
in accountability would be for the ARC to publish each year statistics 
reporting not only the level of consensus between assessors who 
evaluated the same application, but the correlation between the rank 
ordering of applications by each ARC panel and the average of the 
ratings assessors provided for these applications. If funding deci¬ 
sions do not correlate highly with the average of assessor ratings, it 
would be incumbent upon the ARC to justify the processes it 
employs and to defend the expertise of the persons it selects as pane! 
members. At present the ARC is subject to limited accountability. 

Under the appeal mechanism introduced in 1991, unsuccessful 
applicants can appeal against decisions made by the ARC “... if they 
feel there were deficiencies in the procedures followed”, but not on 
the basis of".... the peer review system itself, or of the judged merit 
of the proposal (ARC News, 1992, p. 4). Since the ARC maintains a 
level of secrecy about its procedures, applicants generally lack the 
information needed to identify “deficiencies in the procedures 
followed”. The only obvious grounds for appeal is that an applica¬ 
tion was culled without being sent to assessors for evaluation. 
Applicants in time receive assessors’ reports (but are not given the 
names of assessors) as well as the rating given to their project by the 
panel. The appeal mechanism seemingly does not allow an unsuc¬ 
cessful applicant to challenge the expertise of the assessors ap¬ 
pointed by the panel, or to question the use that the panel made of 
reports received from assessors. Further, more general questions 
relating to peer review, such as the level of consensus in ratings 
between assessors and the possibility that selecting persons other 
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than the assessors appointed by the panel would have produced a 
different outcome, cannot be raised. On the grounds of accountabil¬ 
ity, the ARC should provide more explicit guidelines on grounds and 
processes for appeal than it does at present. 

It is only recently that the ARC has made serious efforts to 
evaluate the impact its funding decisions have had on research and 
scholarship in different disciplines. Retrospective reviews, covering 
a five year period, are being undertaken by experts within the 
discipline who themselves were not eligible for ARC support. 
Reviews have been completed in economics, British and European 
history, Australian history, genetics and evolution, igneous and 
metamorphic petrology and geochemistry, condensed matter phys¬ 
ics, mathematics, and ecology, while reviews are either under way 
or planned in materials and chemical engineering, organic chemis¬ 
try, fluid mechanics, molecular biology, experimental physics, and 
psychology. While such reviews are useful in identifying “hits” 
(projects that were funded with a successful outcome, as assessed by 
indicators such as publications and citations) and “false alarms” 
(funded projects that proved non-productive), the extent to which the 
ARC decision making processes resulted in “misses” (failure to fund 
projects that would have had a successful outcome) is difficult to 
establish. Simply comparing the productivity and impact of appli¬ 
cants supported by ARC with measures for those who were not 
supported does not address this question, since in many disciplines 
the ARC is the only source in Australia for funding basic research. 
Rejection of an application by the ARC often means that the project 
cannot proceed, even though the research may have had successful 
outcomes if it had been funded. The retrospective reviews being 
undertaken by the ARC need to focus not only on hits (successful 
outcomes), but on documentation of the consequences of failure to 
fund projects. 

Further comments 

As well as being valid, the procedures followed by the ARC in 
evaluating applications must be cost effective, in order to reduce 
administrative costs, the ARC in 1990 required each applicant to 
nominate two assessors, who provided reports directly to the ARC. 
This system operated for only one year, presumably because the 
evaluations that came from applicant-nominated assessors were 
generally too favourable and did not prove to be useful. Although 
applicants are now asked to provide a list of potential assessors, all 
persons requested to evaluate an application are selected by ARC 
panels. As a further experimental approach, the ARC in 1993 
requested assessors not only to provide global evaluations, but to 
rate the project and the applicant in terms of specific criteria. Since 
there has been disagreement (see Gottfredson, 1978; Marsh & Ball, 
1989) as to whether ratings of specific attributes offer a more 
effective basis for evaluation than global ratings, it is to be hoped 
that the ARC will analyse and publish the data it collected on this 
issue. 

Other ARC procedures have changed over time. For several years 
all applications in plant molecular biology, gene regulation, animal 
hormones, and ecology were evaluated by small sub-panels without 
applicants being interviewed and without external peer review. 
Since a system of specialist sub-panels is a feasible alternative to 
multi-disciplinary panels (where many applications may lie outside 
the expertise of any panel member), it would be interesting to know 
whether the ARC has evaluated (on an effectiveness as well as cost 
basis) the use of specialist rather than generalist panels. Related 
questions, particularly when generalist panels are used, are whether 
it is preferable to rely on few assessors each rating many applications 
rather than many assessors each rating few applications and whether 
assessors should be given some training in order to foster standardi¬ 
sation in evaluation. However, there is a limit to the demands the 
ARC can place on assessors, who serve without payment (they once 
received an honorarium). 

The ARC and its predecessor, the Australian Research Grants 
Council, employed for many years a costly system of interviews. In 


1992, for example, almost 800 ARC applicants were interviewed in 
nine cities. Although the seeming objective was to allow ARC panel 
members to explore issues raised by assessors, applicants could not 
respond directly to comments made by assessors, since it was only 
after funding decisions had been made that assessor reports were 
released to applicants. At least to applicants, interviews often 
seemed like a “fishing expedition”. In view of research on the extent 
to which expectations and irrelevant information can bias impres¬ 
sions and distort judgments (see Millar, Crute, & Hargie, 1992), it 
is disappointing that the extent interviews influenced ARC funding 
decisions was never subjected to scrutiny. In 1994 the ARC, for the 
first time, will not interview applicants. Instead the 40% of appli¬ 
cants surviving the July cull will be sent assessors’ reports, and be 
able to provide the panel with a one-page response. This process, 
which does not extend the review period and is much more cost 
effective than interviews, will permit applicants to respond to 
assessors in a more informed and considered manner than they could 
through the interview system. Further, by receiving a written re¬ 
sponse from an applicant, the ARC panel is likely to be more 
informed as to whether it should obtain additional expert opinion. 

The changes in the Australian higher education system that led to 
“clawback” and greater central control over the allocation of re¬ 
search funds are unlikely to be reversed. Although a substantia! 
proportion of all basic research in Australia is undertaken within 
universities, these institutions themselves are unlikely to gain the 
level of control over the distribution of funds that they once had. One 
means of indirect influence is by encouraging academics to seek 
grants. In commenting on funding under the ARC Large Grant 
Scheme, Brennan, the ARC chair, argued that the 19% success rate 
for funding under the Large Grants Scheme in 1993 reflected 
excessive demand for support rather than an inadequate level of 
funding (“the current level of government supported activity in basic 
research in Australian universities is about average for the middle 
sized OECD countries”) (ARC News, 1992, No. 4, p. 1). Brennan’s 
claim is that ARC Large Grant Scheme applications rose dramati¬ 
cally because of the emphasis placed on research funding as a 
performance indicator, and an index of prestige, within the National 
Unified System. In commenting on replacement of the binary system 
(where universities were funded for research but colleges of ad¬ 
vanced education were not) by the National Unified System, Meek 
(1991) suggested, "There is a mystique attached to the very name 
‘researcher’, and to be denied access to the title will lead to two 
classes of institutions and two classes of academics within institu¬ 
tions; researcher and non-researcher, with status, prestige, and 
wealth allocated accordingly....If status, wealth, and prestige are 
directly linked with research, with little or no avenue to gain an 
advantageous position in the hierarchy without a research function, 
then the result probably will be rampant institutional imitation” (p. 
477). Since applications for funding under the ARC Large Grants 
Scheme increased by 18% in each of 1991 and 1992, such seems to 
have been the case. 

Brennan, writing in his capacity as ARC chair, argued not that 
more funds need to be available for basic research, but that the 
demand for support through the ARC Large Grants Scheme needs to 
be dampened (“one must ask whether steps are needed to avoid 
further increases in demand for ARC funds”) (ARC News, 1992, No. 
4, p. 2). It will be interesting to establish the longer-term effects of 
the recent ARC rejection rates. The outcomes of interest are not only 
the number and nature of applications in the future but the effects 
that failure to receive ARC support will have on the morale and 
productivity of academics, many of whom have a record of research 
achievement. A critical issue is whether, in deciding which appli¬ 
cants to support at a time where the demand for funds grossly 
exceeded the supply, the ARC panels made the best possible deci¬ 
sions. An underlying question is whether the processes employed by 
the ARC are those likely to distribute research funds with most 
effect. 
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Conclusions 

Although ARC panel members who decide which applications 
under the Large Grants Scheme will be funded have access to peer 
review, the extent to which funding by the ARC reflects peer review 
is uncertain. It is difficult to find information on how ARC panels 
operate, let alone obtain statistics. Public debate on the processes by 
which the ARC distributes research funds has been restricted by 
limited access to information. My primary objective in preparing 
this article is to urge the ARC to publish a detailed outline of how 
ARC panels operate at different stages in the decision making 
process (e. g., selection of assessors, culling, use of assessors’ 
reports, ranking of applications). Further, since the data are rou¬ 
tinely coded for computer entry and analysis, it would be a simple 
matter for the ARC to publish each year statistics bearing on peer 
review (e. g., level of consensus between assessors in different 
disciplines) and decisions made by panels (e. g., level of agreement 
with peer review). 

Placing information of the above kind within the public domain (in 
the past it has been collected, but distributed only within the ARC) 
will allow informed debate about the processes adopted by the ARC. 
Further, the ARC will be subjected to demands for accountability 
that do not apply at present. For example, if it turns out that funding 
decisions correspond with evaluations provided by assessors, issues 
such as the selection of assessors and the criteria assessors employ 
in evaluating applications are of vital importance. Attention would 
instead focus on the selection of panel members and the manner in 
which they operate if it is shown through statistical analysis that 
members of ARC panels have substantial independent influence on 
funding decisions. The processes employed by the ARC have not 
been open to external scrutiny, and the ARC has never been required 
to justify the procedures it follows. In the spirit of accountability, the 
ARC should accept responsibility for providing the information 
needed to permit informed evaluation of its processes and effective¬ 
ness. 
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